#EvenRealitiesG2 #SpeakerRecognition #VoiceEmbedding #LocalSTT #RAG #SmartGlasses

This is how we built Even G2 speaker recognition and automatic memory storage into a conversation recorder of our own. The microphone audio from the Even Realities G2 glasses goes to a server we run, which transcribes it and labels each line as the wearer, a numbered attendee or a named attendee. When a conversation ends, a local model summarises it and files the summary in a personal memory bank that our assistant bots can search. The storage half is running: as of 2026-11-03, 21 of 22 summaries are marked as stored, all 21 could be matched in the memory database by conversation start time, and they hold 142 facts, though we added 3 of the 21 by hand. The speaker half is less settled. Two people’s voice vectors overlapped too much for one threshold to separate them, so we moved to regrouping the whole conversation, naming groups instead of lines, cutting lines before transcription, and showing a label only when the measurements support it. Only three lines have been checked by a person, and one of those is still wrong. Line cutting has been through replay tests but no real meeting.

🔍 Why not use what ships with the glasses

The G2’s own conversation feature, Conversate, gives live captions and summaries. When we checked on 2026-10-01 it kept one stream with no speaker labels, and the record stayed inside the Even app. A third-party review also notes the missing speaker labels. A forum post says labels are planned, but it is secondhand and we could not find the original. We wanted two things Conversate did not give us: who said what, and a record on our own server that other tools could use.

Other people have built on the same glasses. This is what we found on 2026-10-01; we re-checked on 2026-11-03 that the repositories still exist, but not what they do now. None of the five documented how it behaves while the phone is locked.

🧰 Hardware and pipeline

The G2 has four microphones, but the Even Hub SDK hands an app a single stream of raw audio samples (PCM): 16 kHz, 16-bit, mono. An app is a web page that the Even app runs in a WebView on the phone. Only the display and input travel over BLE to the glasses, so our app does no audio work. It forwards each PCM chunk over a WebSocket as soon as it arrives. We measured 1,600-byte chunks about every 50 ms, or 32,000 bytes per second. In the first 189.5 s session the server received 98.75 % of the expected bytes. After the phone was locked, chunks kept arriving at 19 to 20 per second for about 35 s. We had planned a 10-minute locked test and did not run it.

The server is a Python service on a Mac mini. Silero VAD v6.2.3 (voice activity detection, which finds where speech starts and stops) cuts the stream into utterances. We call each utterance a line, because that is how it appears on the phone. A speaker-embedding model turns audio into a fixed-length vector meant to sit close to other vectors of the same voice; we use CAM++ at 512 dimensions, on one CPU thread, and call its output a voice vector. A 3 s window takes 18.3 to 21.7 ms on average. Transcription goes to a relay on a DGX Spark, currently running Qwen3-ASR-1.7B (we wrote up that engine swap separately), and a cloud engine can be chosen on the phone at the start of a session. Lines go into PostgreSQL. Audio stays on an external SSD for 90 days, so tapping a line on the phone plays it from that point.

We chose CAM++ on published numbers alone. WeSpeaker’s results list a base equal error rate (EER, the point where false accepts and false rejects are equally common) of 0.803 % on VoxCeleb1-O, against 1.069 % for ECAPA-512. ECAPA-512 ran about three times faster on the Mac mini. We never measured either model on the wearer’s voice or on Korean, and we did not find a published Korean figure for any candidate.

The SDK does offer its own hint. Each audio frame carries a speakerRole of self, other or unknown, and the SDK’s comment says this is not a firmware identity decision. Over about 75 s we counted self 145, other 1,355, unknown 0, and self also rose when something was handled near the glasses. It looks like a measure of sound energy close to the frame, though that is a working guess we have not quantified. We use it only as a side signal.

The running totals as of 2026-11-03: 37 conversations from 2026-10-02 to 11-02, 6.6 hours in all (our sum of their lengths), 2,817 lines, of which 2,615 have text and 2,270 have a voice vector.

📉 Why one threshold failed

The wearer has no enrolment session. The voice profile is confirmed after five lines tapped as “me” across at least two conversations, or ten lines of 2 s or more, each with the SDK’s own-voice flag covering at least 60 % of the line, across three conversations. The tap route exists because in the first trial only 1 line in 34 passed the SDK condition. We also cancelled a planned test in which people would read a script, so we have no verified answer set. That shapes everything below.

A 176.4 s recording of two other people, 19 lines, showed the core problem. Whole-line Cosine similarity (1 means two vectors point the same way, and lower means less alike) ran 0.60 to 0.83 between lines of the same person and 0.51 to 0.80 between the two people. Those ranges overlap almost entirely. Six of the 19 lines held both voices, because the line ends at a silence and not when the speaker changes.

Attendee numbering had a related flaw. Each new line was compared with a running average for each person and joined it at a similarity of 0.5. A five-person meeting came out as TERRY (the on-screen label for the wearer), P1 and P2, and P2 was two people. In a later meeting six or seven people ended up in P1.

👥 Regrouping, then naming groups

We dropped the running average. Now every voice vector in the conversation is clustered again at each new line, and once more when the conversation ends, using average linkage: two groups merge while the mean similarity across all their pairs is at least 0.6. A group of fewer than three lines is not treated as a person and shows ?. On the same five-person meeting, 237 lines became groups of 105, 70 and 53, with 9 left as ?. The group sizes matched scipy’s average linkage cut at distance 0.4, and the run took 11 ms on the Mac mini. The fifth person never separated and stayed inside one of the three groups.

Names came on 2026-11-02. The wearer can name a line on the phone, and the voice vectors of named lines are averaged into a reference for each person. The system then compares a group’s centre, not a single line, with every reference. A name is applied only when the best match is at least 0.75 and leads the second by at least 0.10; otherwise the group keeps its number. A reference needs five lines from at least one conversation, each at least 1 s long. Names the machine assigned never feed back into a reference, because one mistake would then contaminate that person’s reference. A name the wearer chose always wins on screen.

We decide on groups because single lines overlap. The table below covers all 175 named lines we had, under pseudonyms X, Y and Z. Each value is a line’s Cosine against a person’s centre; p10, p50 and p90 are the 10th, 50th and 90th percentiles. For same-person values the centre excludes that line’s own conversation. Conversation b holds 162 of the 175 lines, so its same-person values rest on a centre built from only one or two lines elsewhere.

Centres are close too: X and Y 0.698, Z and Y 0.659, X and Z 0.52. This table does not justify a threshold. It shows only that, line by line, one person’s p10 overlaps another’s p90.

A replay over stored vectors (24 conversations, 1,061 lines, five named groups) gave 4 groups named correctly, 0 wrongly and 1 left unnamed at thresholds from 0.70 to 0.80, and margins from 0.02 to 0.10 did not change that. The names being tested were themselves applied by group, so this is not an independent answer key. In use so far the feature has named 17 lines in one conversation with one name, and nobody has checked whether that name is right. As of 2026-11-02 the name on the lens had not been seen on real glasses.

✂️ Cutting a line before transcription

Regrouping cannot fix a line that holds two voices, because that line has one vector and it falls somewhere between the two people. Meeting 1, one of four recorded meetings we later reprocessed (1,785 s of recording, 181 lines), showed how much this mattered. The wearer threshold was then 0.60, checked before anything else, so mixed vectors and attendees who sound like the wearer were labelled TERRY and never reached grouping. Long stretches make it worse: speech is cut only at a 12 s cap, at the quietest point in the 2 s before it, so one line can hold several people.

The obvious fix is to split the text afterwards using word timings. Our relay returns no words or segments, even when asked with verbose_json and timestamp_granularities. So we cut the audio first and transcribe each piece. For a line of 6 s or more:

1. Slice it into 3 s windows with a 1 s hop, drop windows quieter than 150 in RMS (root mean square, a measure of loudness, on int16 samples), and compute a voice vector for each.
2. With fewer than four windows, stop.
3. At every boundary, take the Cosine between the mean vector of all windows to the left and all windows to the right, and pick the boundary with the smallest value.
4. If that value is 0.4 or more, stop.
5. Cut at the midpoint between the end of the left window and the start of the right one, unless that is within 2 s of either end. The position can be off by about a second.
6. Repeat on each side, to depth 2.

This is the function that does steps 2 to 6, with lines wrapped for this page:

def _cuts(windows, cfg, np, piece_start, piece_end, depth):
    if (len(windows) < cfg.split_min_windows
            or depth > cfg.split_max_depth):
        return [Span(piece_start, piece_end)]
    best = None
    for k in range(1, len(windows)):
        left = np.mean(np.asarray(
            [w.vector for w in windows[:k]],
            dtype=np.float64), axis=0)
        right = np.mean(np.asarray(
            [w.vector for w in windows[k:]],
            dtype=np.float64), axis=0)
        score = cos(left, right, np)
        if best is None or score < best[0]:
            best = (score, k)
    score, k = best
    if score >= cfg.split_threshold:
        return [Span(piece_start, piece_end)]
    cut = (windows[k - 1].end_s + windows[k].start_s) / 2.0
    if (cut - piece_start < cfg.split_min_piece_s
            or piece_end - cut < cfg.split_min_piece_s):
        return [Span(piece_start, piece_end)]
    return (_cuts(windows[:k], cfg, np,
                  piece_start, cut, depth + 1)
            + _cuts(windows[k:], cfg, np,
                    cut, piece_end, depth + 1))

The second half of the change is what the screen shows. A label now needs measured support. Consistency is the 10th percentile of the Cosines between windows that start at least 3 s apart, or, if no such pair exists, the smallest Cosine between a window and the mean. A line under 1 s shows ? before any of this. A mixed line stays out of the wearer’s profile, the name references and the grouping. Pieces of a cut line are also never automatic candidates for the wearer’s profile.

We picked these values from the same recordings, so they should be read as starting points. For window length we compared the wearer-minus-attendee similarity between control lines and lines grouped as one attendee, using the area under the ROC curve (AUC, where 1.0 is perfect separation). The controls are lines with wearer similarity of 0.90 or more, which we assume are the wearer alone. Attendee A’s lines may also contain mixing, so these figures are only indicative. Our 3 s window with a 1 s hop is not itself a row here.

For the cut threshold, we tried six values on meeting 1’s 142 lines of 6 s or more, against its 47 control lines. At 0.4, 14 lines were cut and no control line was.

The wearer threshold rose from 0.60 to 0.80. In meeting 1, pieces of 3 s or more fell into two clear groups by wearer similarity: 40 pieces below 0.8 (5 under 0.3, then 10, 9, 5, 4 and 7 in steps of 0.1) and 128 above it (4 in 0.8–0.85, 35 in 0.85–0.9, 89 in 0.9–1). That split appeared only in meeting 1. In the other meetings, 0.85 cut through the middle of what looked like the wearer, which is why we settled lower. We did not check whether the time lost at each step was really the wearer talking. The table gives seconds labelled TERRY, before the under-5-second rule was added.

📊 Results on four meetings

We ran two kinds of test. The first reprocessed four recorded meetings read-only. The second restored a dump of the production database into a separate PostgreSQL instance and fed recordings in real time, with summaries, tips, search and location turned off: five sessions of 726 s, 915 s and three of 180 s. Unit tests stand at 1,682 passed, 35 skipped and 5 failed; the same 5 fail on the baseline commit.

Reprocessing shows how often lines get cut. Controls are the same assumption as above, and we did not check whether the three control lines cut in meeting 3 were errors or real mixes.

The more important question is how much time ends up with a useful label. “Total” here is the summed length of the pieces, so it leaves out anything outside a line and is shorter than the recording. Figures look rounded (we did not compare raw values), and three rows differ from their total by 1 or 2 s. “Consistent, no name” means the voice held steady but passed no name. “Not judged” means fewer than three windows. This table adds the under-5-second rule, and that is probably the main reason its TERRY times exceed the 0.80 column above, though we inferred that rather than measured it.

Two rows name nobody at all, and in meeting 4 half the time goes unjudged. These numbers predate moving the name decision into shared code and were not re-measured afterwards; the cut counts were, and they came out the same.

Being cautious cost TERRY labels. Lines under 5 s now keep TERRY only at 0.80 or more:

Across all lengths in meeting 2, of 57 lines previously TERRY, 34 kept the label, 13 lost it to the short-line rule, 7 to the 0.80 threshold and 3 to cutting. In the replay of the same recording, 35 of 91 lines were TERRY. We do not know whether the lines that left were really the wearer’s.

The only human check so far is three lines the wearer pointed out. All three had been shown as TERRY.

Line 2 is still wrong. In this replay the speaker changed near the end of the line, so only the last window held the other voice, and the smallest Cosine was 0.5897, above the 0.4 needed for a cut. With the stored line’s boundaries the same speech did get cut. We have not worked out why the boundaries differ between runs. Three lines cannot give an accuracy figure.

Latency is the time from the start of speech to publishing a line, minus the length of the speech. Lines that were not cut did not get slower. Cut lines took about 0 to 1.2 s longer than uncut lines of similar length.

As of 2026-11-03 the server is on 0.21.0 and the app on 0.11.2 (Even Hub Beta). No stored production line carries a consistency score yet, because the last real conversation ended the day before the deploy. Behaviour in a real meeting and the labels on the phone screen are both still unobserved.

🧠 Summaries into memory

Transcripts stay in the conversation database and are never sent to the memory API. Only summaries go to the memory bank. Had transcripts gone into our shared RAG index, every bot able to search that index could read them. Bots that need exact wording can still read the transcript table directly when their rules tell them to, so this separates search from direct reads; it does not make transcripts private from those bots.

The summary runs on Gemma4-26B-A4B on the DGX, served by llama-server with a 131,072-token context and two slots, so the transcript stays on our network. Temperature is 0, thinking is off, the time limit is 120 s, max_tokens is 1,500, a transcript over 60,000 characters is reported as an error rather than cut, and a conversation needs at least three lines of text. It has three steps:

1. One pass over the whole transcript writes only “what was discussed”.
2. Windows of 70 lines, overlapping by 10, extract decisions, to-dos and requests made of the wearer, each as {"text", "at"}. An item whose at is not a line in its window is dropped and counted as uncited.
3. The numbered candidates go back to the model, which may answer only with groups of numbers that mean the same thing. A group keeps its first item’s text, and a candidate in no group stays. The reply is constrained by a json_schema. This step can neither invent an item nor remove one.

The windows came from a failure. One pass over a 64-minute conversation (411 lines) missed requests, because the to-do rule accepts only what the wearer said, and it returned an empty list of decisions. Windows recovered the 4 missed items, 3 requests and 1 decision. Time went from about 8 s to 50.7 s, over 7 windows with 0 uncited. The result also has more noise: 14 requests and 10 decisions, some of them the wearer’s own remarks or things said only as possibilities. We decided a missed request costs more than a line of noise, which is why the last step cannot drop anything. A single pass over a 280-line conversation (15,519 characters) took 6.2 s.

When a conversation ends, the server posts the summary to the memory API on the same host:

{"content": "<date, start and end, speakers,
   four sections, conversation id>",
 "bank": "<wearer's personal bank>",
 "tags": ["g2-conversation", "summary"],
 "timestamp": "<conversation start>",
 "source_env": "g2"}

The memory server splits this into facts in the background and records the conversation’s start as occurred_at. An LLM does that split as well, so some facts come out garbled. The key for this call can reach only memory/retain and that one bank; a request to another bank or to memory/recall got a 403 when we tried. A failed store records its cause and is not retried, and the app shows the state. A summary regenerated by hand is stored only when the wearer presses a button, and the screen warns that storing twice makes two memories.

Our assistant bots have a rule in their system prompts: search that bank’s g2-conversation tag first, and read the transcript table if exact words matter. We added it on 2026-10-23. Before that, a bot could search only its own bank and report that no record existed.

An API accepting a request does not show the data landed, so we compared the two databases directly on 2026-11-03:

Three of the 21 were added by hand, for the reason given in the next section. There are 15 conversations with no summary; we did not count how many were too short and how many predate the feature.

Data also flows the other way. The tips on the lens can draw on the same RAG’s document search. A small router model sends keywords only, never transcript sentences. Fifteen work keywords gave latency of p50 1.25 s, p90 3.46 s and a maximum of 6.42 s. For 9 of them the top rerank score (a second, more careful relevance scoring of the search results) was under 0.02 and the document was unrelated, and the 5 real matches scored 0.20 to 0.96. The record does not give the remaining keyword’s result. Evidence under 0.2 is thrown away, and a tip appears only if the passage it quotes is present in the fetched evidence, which code checks; the check lets a quote skip between fragments as long as their order holds. Tip quality itself is outside this article.

⚠️ What went wrong along the way

Some of these reached production and some were caught before deploy. We say which.

• Summaries depended on Stop (production). A summary ran only if the phone sent Stop and received the server’s stopped frame. Three conversations of 30 to 60 minutes ended with the app closed or the network gone, and they got neither summary nor memory. Since 2026-10-23, any conversation with enough lines gets a summary however it ends. We summarised those three by hand.
• A shared variable name (before deploy). The key’s proposed environment variable already held a search-only key for another feature. Every Stop would have stored a 403. We caught it just before release.
• An overlay that replaces (setup). A list in the config overlay replaces the base file’s list instead of extending it, so the host’s whole key list moved into the overlay.
• Thinking ate the answer (integration). The summary model thinks by default. A request for a 60-token answer spent all 60 tokens thinking. The request now turns thinking off.
• A merge step that rewrote (pipeline). Asked to rewrite items, the merge passed the 1,500-token limit. Asked to drop vague items, it dropped the most specific request. It now answers with numbers only.
• Loose JSON (parser). The model answered todos as a list of objects and the parser rejected it twice, which led to the json_schema.
• Engine failover erased voice vectors (production). When the DGX stalled once, the session moved to a backup engine and returned within 7 s. On return the audio ring restarted at sample 0 while the shared segmenter’s clock stood near 20 million samples, so ring lookups failed until the session ended and 59 lines were stored without a vector. We removed failover on 2026-11-02. A session keeps the engine chosen on the phone. After three straight failures, transcription pauses, the same engine is probed, the cause appears on the phone, and transcription resumes when the engine recovers.
• Whisper repeating its hints (STT, production). Given a glossary prompt, whisper sometimes returned the end of the prompt as the transcript of a session’s first line. On 30 public clips, attenuated, this hit 2 at −10 dB, 6 at −20 dB and 7 at −30 dB, and none without the glossary, so the glossary went. Separately, a 0.6 s fragment transcribed as one repeated word entered the context and turned the next 10.3 s and 11.0 s lines into that word. Such lines no longer enter the context.
• Three defects a review missed (before deploy). The review looked at tests and feeds that all used an engine able to transcribe pieces. Playing recordings end to end found them: an engine that delivers a line’s text with its final event put the same text into every piece; a whole-line SDK flag let another person’s piece into the wearer’s profile candidates; and a piece’s vector was lost when its audio had already left the ring. After the fixes, replays with openai-live for 180 s and dgx-asr for 180 s twice showed no duplicated text. The second defect’s fix was checked by unit test only, because the feeder sends no SDK flag.
• Old app, new server (compatibility). An old app marking a cut line sends only the line number, which could have applied to piece 0 and put someone else’s vector into a name reference. The server now refuses it with a 409.

🛑 The case against this design

We cannot state its accuracy. One wrong line out of three checked is not a rate. Without the scripted test there is no answer set, and values are tuned from the wearer’s corrections. Until enough of those accumulate, we cannot put a number on any of this.

Names rest on very little. With a minimum of one conversation, the sound of one room becomes that person’s reference. The value can be raised, but 162 of our 175 named lines come from a single conversation today.

Caution loses information. Meeting 4 leaves 246 of 491 s not judged, and meeting 2’s TERRY lines fell from 57 to 34. Fewer TERRY lines can also mean fewer to-dos in the summary. We ran the summary three times on each of two transcripts of one meeting and got means of 5.67 and 3.33 to-dos, but the transcripts’ text differs, so that says nothing about the labels.

Cutting depends on the engine. Engines that deliver text with the final event cannot transcribe pieces, so only mixed marking runs there.

Consistency can flatter noise. Two pieces with no transcribed text scored 0.85 and 0.98, which suggests silence or noise can look consistent. One is a 5.0 s front piece of a line. The other is the only line handled by a different engine, and its audio position may have been off; we did not confirm that.

Some attendees sit on the line. Attendee B’s confirmed line has wearer similarity 0.78, just under the threshold, and some pieces that look like the wearer score 0.76 to 0.79. Both sit in the same band, so a fixed threshold cannot separate them.

Someone else’s product may do better. Cloud diarization such as Deepgram’s, which Cue uses, needs none of this code. We did not test it. Nor did we evaluate a dedicated model such as NVIDIA Sortformer. We have no comparison in either direction.

🎯 What to take from it

These are the choices we made and what we would check first in your place.

• Decide where transcripts live before wiring up memory. We pass only summaries to the searchable store, so what a store holds and who can reach it can be handled separately.
• Check storage in the store. Our “done” means matching records in the memory database. We also stopped tying summaries to a clean Stop, after three long conversations ended without one.
• Suspect mixed lines before tuning thresholds. If your transcriber gives no word timings, cutting the audio first is the route we found. Check whether your engine can transcribe pieces at all.
• Without an answer set, label cautiously and log corrections. We show ? and mixed rather than a confident guess, and we accept the lost time that costs.

If you only want notes from your own short conversations, start with the SDK’s signal and a cloud transcriber and see whether that is good enough; we did not measure this path. For multi-person meetings and a long-term record, we ended up needing regrouping, group naming, line cutting and a correction screen together, and the line-cutting latency is in the table above. Either way, every number here comes from one wearer and a handful of meetings.

📚 References

• Even Hub documentation
• Even Realities Conversate
• HearingTracker forum review of the G2
• 3D-Speaker (CAM++)
• WeSpeaker VoxCeleb results
• Silero VAD
• Our earlier post on the speech-to-text engine swap
• Prior projects: whisperlivekit-translate, Cue, Cue-evenRealitiesG2, even-reality-memory-system, open-g2-hub