Early Get Smart attempts at categorization

An Overview of Scenerios Already Completed on Eckford

1. AI/ML-adjacent pipeline development

We built a multi-step processing pipeline around recipes.

The pipeline moved roughly like this:

Backdrop CMS recipe content
        ↓
CSV export
        ↓
Python import / normalization
        ↓
recipe JSONL
        ↓
runtime recipe catalog
        ↓
ingredient extraction
        ↓
nutrition input filtering
        ↓
nutrition lookup
        ↓
runtime-ready recipe data

The important part: we separated the work into stages.

Each stage had a job:

Source Door              [DONE]
Retrieval Door           [DONE]
Classification Door      [DONE]
Nutrition Input Layer    [NOW BUILDING]
Nutrition Contract Door  [READY]

That “doors” idea mattered because it prevented the project from becoming one giant soup pot.

Each door answered:

Do we have source data?
Can we retrieve it?
Can we classify it?
Can we prepare nutrition inputs?
Can we define the contract for nutrition output?

This is pipeline thinking. Very ML-adjacent because before any model or calculation can work, the input data has to be shaped, filtered, and validated.

---

2. Data normalization

The recipe data started in Backdrop CMS.

The exported data was not naturally “machine clean.” It had fields like:

Recipe
Content
Dish
Stage
food_pics
Season
status
Post date

We used Python scripts to convert that into normalized runtime files.

The key files were something like:

recipes.csv
recipes_normalized.jsonl
runtime_recipes.json
runtime_build_summary.json

The normalization goal was:

Keep source content unchanged.
Extract what we need.
Flag what is missing.
Build clean runtime data.

So instead of editing the original recipe text directly, we created structured derived files.

That was a good design choice.

It meant:

Backdrop remains source of truth.
Python processing creates machine-readable outputs.
Problems become flags, not silent failures.

Very Get Smart. Very “don’t lie to the machine.” 🕵️

---

3. Classification gates

This was the big one.

We realized that not every ingredient line should be used for nutrition calculation.

A recipe might contain:

salt
pepper
basil
oregano
garlic powder
1 tablespoon olive oil
2 cups chickpeas
1 pound pasta

For nutrition, some lines matter more than others.

So we created the concept of classification gates:

candidate ingredient line?
        ↓
primary ingredient?
        ↓
herb/spice/seasoning?
        ↓
nutrition-relevant?
        ↓
send to lookup or skip?

The classification gate separated:

primary ingredients

from:

herbs
spices
seasonings
minor flavorings

That gave us this rule:

Nutrition input is derived from ingredient lines by:
- filtering candidate lines
- classifying herb/spice/seasoning vs primary
- selecting primary lines only
- passing normalized lines to deterministic lookup

Source data remains unchanged.

That is the heart of the work.

It was not just “calculate calories.”
It was decide what deserves to be calculated.

That is very close to applied ML thinking, even if the classifier was not yet a trained scikit-learn model.

---

4. Vector retrieval

We also worked with vector/retrieval concepts.

The Chroma/vector side was there to help with recipe and nutrition search. The important architectural decision was:

Chroma can help retrieve or compare information.
Chroma is not the source of truth for nutrition.

The source of truth was the structured nutrition dataset, especially:

eurofir_mediterranean.csv

That distinction mattered.

Vector retrieval is fuzzy and useful. Nutrition calculation needs traceability.

So the pattern became:

Use retrieval to help find candidates.
Use deterministic lookup for final nutrition values.

That is exactly the right instinct for this kind of system.

No “the embedding felt hungry so it guessed 400 calories.” Absolutely not. Fork confiscated. 🍴

---

5. Agent/tool integration

We also had an agent/tool layer around the project.

The tools/frameworks in play included:

Python
Chainlit
OpenAI Agents SDK
Chroma / chromadb
Flask viewer or admin pages
CSV / JSONL / JSON runtime files
recipe_catalog.py
nutrition_lookup.py
nutrition_calculator.py

The agent idea was not “let the AI invent nutrition.”

The safer design was:

Agent can help inspect, retrieve, explain, or route.
Tools do the deterministic work.
Structured data provides the answer.

That is a good architecture.

The agent is the helpful clerk.
The nutrition calculator is the ledger.
The source CSV is the law book.

Tiny bureaucracy, but useful. 🧾

---

6. Nutrition calculation from structured data

The nutrition calculation piece depended on turning recipe ingredients into lookup-ready inputs.

So the pipeline needed to know things like:

What is the ingredient?
How much is there?
Is the unit usable?
Is this a primary ingredient?
Can it be matched to nutrition data?
Should it be skipped or flagged?

The calculation depended on structured records, not raw prose.

The nutrition system was moving toward this kind of contract:

normalized ingredient line
quantity
unit
ingredient name
classification
lookup candidate
matched nutrition record
confidence / review flag
calculated nutrition values

And when the recipe was not ready, it got flagged.

Examples of skip/review reasons included:

missing ingredient section
insufficient structured ingredients
possible truncation
not recipe full
recipe partial

That was a major piece of the work: not pretending every recipe was ready.

The build summary gave a reality check:

runtime-ready recipes
skipped recipes
missing ingredient sections
partial recipes
review-needed items

Again: evidence first. The machine does not get to swagger.

---

The actual achievement

The main thing you did was not “train a recipe model.”

You built the foundation that a model would need.

That foundation includes:

clean recipe exports
normalized runtime data
ingredient extraction
classification rules
nutrition-relevant filtering
deterministic nutrition lookup
review flags
runtime summaries
agent/tool architecture
vector retrieval support

That is exactly the kind of pre-ML work that decides whether an ML system succeeds or turns into a glittering garbage barge.

The best plain-English summary

We built a Python-based recipe intelligence pipeline for Get Smart.

It took messy recipe content from Backdrop CMS, normalized it into structured runtime files, identified usable ingredient lines, separated primary ingredients from herbs and seasonings, and prepared clean nutrition inputs for deterministic lookup against a European nutrition dataset.

The system preserved the original source content, produced review flags when recipes were incomplete, and used retrieval/vector tooling and agent-style interfaces as support layers rather than as the final authority.

Where this can go next

The next “Yes” step is to add a small, real ML layer on top of this existing foundation.

The best candidates are:

1. scikit-learn classification
   Predict whether a recipe is nutrition-ready or review-required.

2. scikit-learn clustering
   Group recipes by ingredient patterns, dish type, or nutrition profile.

3. scikit-learn regression
   Estimate calories or nutrition bands from structured ingredient features.

4. dimensionality reduction
   Visualize recipes based on ingredient/nutrition similarity.

For Friday/Saturday, the cleanest first bite is probably:

Build a scikit-learn classifier that predicts:
nutrition_ready vs review_required

Because you already have the raw material:

recipe fields
status
stage
dish
ingredient structure
skip reasons
review flags
runtime-ready status

That would turn your existing Get Smart work from:

AI/ML-adjacent pipeline

into:

hands-on scikit-learn classification project

That is the bridge. The planks are already stacked beside the river.

---

Pick it up again for Friday

---

Yes. This is a good “ease back in” path. The hat rack is allowed. We just need to stop letting the ML hat gather dramatic dust. 🎩⚙️

For Get Smart, I would learn scikit-learn in this order:

1. What scikit-learn is good at
2. The basic vocabulary: X, y, fit, predict
3. Classification
4. Preprocessing and pipelines
5. Evaluation
6. Clustering
7. Dimensionality reduction
8. Regression

Not all at once. Tiny bites. No buffet injuries.

1. What scikit-learn is good at

scikit-learn is a Python machine learning library for predictive data analysis, built on NumPy, SciPy, and matplotlib. It is especially useful for classic ML tasks like classification, regression, clustering, preprocessing, model selection, and evaluation. (scikit-learn)

For your purposes, it is probably better than starting with TensorFlow or PyTorch because scikit-learn is more direct for:

tabular data
small-to-medium datasets
classification
regression
clustering
feature engineering
evaluation reports

Your Get Smart data is exactly that kind of creature: recipe rows, ingredient counts, flags, categories, stages, status fields, nutrition fields.

TensorFlow and PyTorch are more “build and train neural networks.” Useful later, but right now they are wearing armor to wash a teacup.

2. The core scikit-learn mental model

The core pattern is:

model.fit(X_train, y_train)
predictions = model.predict(X_test)

That’s the little spellbook.

In plain English:

X = the evidence / input features
y = the answer you want the model to learn
fit = learn patterns from examples
predict = apply those patterns to new rows

For Get Smart:

X could be:
- number of ingredient lines
- has ingredient section: yes/no
- dish type
- recipe stage
- number of primary ingredients
- number of herbs/spices/seasonings
- missing quantity count
- matched nutrition records count

y could be:
- nutrition_ready
- review_required

That is your clean first ML target.

3. First thing to learn: classification

Start with classification.

Classification answers:

Which bucket does this belong in?

For Get Smart, examples are:

nutrition_ready vs review_required
primary ingredient vs seasoning
recipe vs non-recipe
complete recipe vs partial recipe
meal type: soup / salad / pasta / beans / dessert

The first practical model should probably be:

Can scikit-learn predict whether a recipe is nutrition-ready?

That is friendly because you already have pipeline status concepts.

A simple first classifier could use:

LogisticRegression
DecisionTreeClassifier
RandomForestClassifier

LogisticRegression is a common starting classifier in scikit-learn, and the current docs describe it as regularized logistic regression for classification, supporting dense and sparse input. (scikit-learn)

My advice: begin with LogisticRegression, then try DecisionTreeClassifier only after the first model runs.

4. Second thing: preprocessing

This is where your existing skills matter most.

Machine learning does not eat raw recipe prose politely. It needs features.

So your real work is turning this:

1 medium onion, chopped
2 cans chickpeas, drained
salt and pepper to taste
fresh parsley

into something like:

ingredient_line_count = 4
primary_ingredient_count = 2
seasoning_count = 2
has_quantities = true
missing_units = 1
nutrition_lookup_candidates = 2

scikit-learn has preprocessing tools for changing raw feature vectors into representations models can use. (scikit-learn)

For Get Smart, you will want to learn these preprocessing ideas first:

numeric features
categorical features
text features
missing values
scaling
one-hot encoding

Do not start with all of them. Start with numeric features.

A “baby dragon” feature table could be:

recipe_id	ingredient_lines	primary_lines	seasoning_lines	missing_qty	matched_items	label
101	12	8	4	1	7	ready
102	3	1	2	3	0	review
103	9	6	3	0	6	ready

That is enough to start.

5. Third thing: pipelines

Pipelines matter because they keep the steps tied together.

scikit-learn’s Pipeline chains preprocessing and a final estimator so you can call fit and predict once on the whole sequence. The docs also describe using Pipeline with ColumnTransformer to combine different preprocessing steps into one feature space. (scikit-learn)

In Get Smart terms:

raw recipe feature table
        ↓
clean numeric/categorical columns
        ↓
preprocess
        ↓
classifier
        ↓
prediction: ready or review

A pipeline prevents accidental nonsense like preprocessing your test data differently from your training data.

That matters. Without pipelines, the gremlins get keys.

6. Fourth thing: evaluation

You do not just ask, “Did the model run?”

You ask:

Was it right?
When was it wrong?
What kind of wrong?
Is it useful?

Start with:

accuracy
confusion matrix
precision
recall
classification report

Accuracy is the simplest metric, and scikit-learn’s accuracy_score measures the fraction of correct predictions. (scikit-learn)

But for Get Smart, accuracy alone is not enough.

Example:

If a recipe needs review, but the model says ready, that is bad.
If a recipe is ready, but the model says review, that is annoying but safer.

So you care about false ready cases.

Your policy brain already knows this. ML evaluation is just the machine version of “what kind of mistake are we willing to tolerate?”

7. Then learn clustering

After classification, clustering is a good second act.

Clustering answers:

Which things naturally group together?

For Get Smart:

Group recipes by ingredient patterns.
Group recipes by nutrition profile.
Find similar dishes.
Spot oddball recipes.
Separate herb-heavy recipes from protein-heavy recipes.

This is where your vector retrieval work connects nicely. You already understand “similarity” from embeddings/Chroma. Clustering is a more traditional ML version of grouping similar rows.

Start later with:

KMeans
PCA visualization
ingredient-count features

No need to lead with clustering. It is interesting, but classification gives you traction faster.

8. Then dimensionality reduction

Dimensionality reduction answers:

Can I compress many features into fewer features so I can inspect patterns?

For Get Smart:

Can I plot recipes based on ingredient/nutrition similarity?
Can I see clusters of soups, salads, pasta, bean dishes?
Can I spot weird records?

Learn:

PCA
possibly t-SNE or UMAP later

PCA is enough at first.

This is more exploratory than operational. Useful, but not the first rung on the ladder.

9. Regression last

Regression predicts a number.

For Get Smart:

estimated calories
estimated protein
estimated prep time
nutrition confidence score

This is tempting, but I would not start here.

Nutrition numbers are fussy. If the source lookup is deterministic, regression may be less useful at first than classification.

A better early regression exercise would be:

Predict a nutrition-readiness score from recipe features.

But still: after classification.

Your first learning plan

Here is the clean path for the next few days.

Friday: Re-entry day

Goal: understand scikit-learn without drowning.

Learn:

X and y
fit and predict
train/test split
classification
accuracy
confusion matrix

Do one toy example from docs or a tiny handmade table.

Then build a tiny Get Smart-style table manually:

recipe_id
ingredient_line_count
primary_line_count
seasoning_line_count
missing_quantity_count
matched_nutrition_count
label

The label is:

ready
review

That’s it.

Saturday: Make it Get Smart-shaped

Goal: build a tiny classifier.

Steps:

1. Load recipe feature table with pandas.
2. Split into train/test.
3. Train LogisticRegression.
4. Predict ready/review.
5. Print accuracy.
6. Print confusion matrix.
7. Inspect wrong predictions.

Then write a tiny note:

What features helped?
What failures happened?
What should remain deterministic?
What should be reviewed by a human?

That last part is what makes this yours.

The first concept set to learn

Start here:

scikit-learn estimator
features X
target y
fit()
predict()
train_test_split
LogisticRegression
accuracy_score
confusion_matrix
Pipeline

That is the starter kit.

Not neural networks.
Not hyperparameter tuning.
Not “let us contemplate the cathedral of gradient descent.”
Just: can we teach the computer to sort recipe records into useful buckets?

Best first Get Smart ML project

I’d name it:

nutrition_readiness_classifier

Purpose:

Predict whether a recipe is ready for deterministic nutrition calculation or needs review.

Inputs:

ingredient_line_count
primary_ingredient_count
seasoning_count
missing_quantity_count
missing_unit_count
lookup_match_count
has_servings
has_title
dish_type
stage

Output:

READY
REVIEW_REQUIRED

Why this is the right first project:

It uses your existing pipeline.
It does not replace deterministic nutrition lookup.
It creates a real scikit-learn classification use case.
It supports your confidence-gated design.
It gives you something concrete to continue.

That is the bite. Not too large. Not trivial. Chewy enough.