Draft: GPT2 training on MCQ med data#1111
Open
mina5rovic wants to merge 60 commits into
Open
Conversation
mina5rovic
changed the title
mina5rovic
force-pushed
the
gpt2-training
branch
from
April 17, 2026 17:53
fd4b676 to
3014cf5
Compare
…k for model to contxt 512
JulienVig
reviewed
Jul 17, 2026
JulienVig
left a comment
JulienVig
left a comment
Collaborator
There was a problem hiding this comment.
1st part of my review, I'll finish at a different time, feel free to start addressing these comments
| /simple_face-example.png | ||
| /titanic* | ||
| /mnist* | ||
| /medicalMCQtxtNoExplanation |
Comment on lines
+20
to
+25
| roundIterations?: number; | ||
| batchSize: number; | ||
| validationSplit: number; | ||
| validationFrequency?: number; | ||
| datasetPath?: string; | ||
| validationDatasetPath?: string; |
Collaborator
There was a problem hiding this comment.
Can you document how all the validation parameters interact with each others? For example what happens if you specify both validationSplit and validationDatasetPath?
| ) | ||
| throw new Error("validationFrequency must be a non-negative integer"); | ||
|
|
||
| // if (!Number.isInteger(this.#epochs / this.#roundDuration)) |
Collaborator
There was a problem hiding this comment.
Suggested change
| // if (!Number.isInteger(this.#epochs / this.#roundDuration)) |
Comment on lines
+159
to
+164
| debugProcessMemory( | ||
| `[${shortenId(this.ownId)}] round ${round} before encode`, | ||
| ); | ||
| const payload = await serialization.weights.encode(payloadToServer); | ||
| debugProcessMemory( | ||
| `[${shortenId(this.ownId)}] round ${round} after encode`, |
Collaborator
There was a problem hiding this comment.
is it useful to log the before after?
| id: NodeID; | ||
| waitForMoreParticipants: boolean; | ||
| payload: serialization.Encoded; | ||
| payload?: serialization.Encoded | null; |
Comment on lines
+148
to
+149
| ? this.#runRounds(dataset, validationDataset) | ||
| : this.#runIterationRounds(dataset, validationDataset); |
Collaborator
There was a problem hiding this comment.
I think these method names are not explicit enough, what about #runRoundsByEpoch and #runRoundsByIteration? Can you add documentation for each method to clearly differentiate their intended use and behaviors?
Collaborator
There was a problem hiding this comment.
Same comment for runRound and runIterationRound
Comment on lines
+208
to
+211
| let pendingBatch: Batched<DataFormat.ModelEncoded[D]> | undefined = | ||
| next.done === true ? undefined : next.value; | ||
|
|
||
| while (pendingBatch !== undefined) { |
Collaborator
There was a problem hiding this comment.
Suggested change
| let pendingBatch: Batched<DataFormat.ModelEncoded[D]> | undefined = | |
| next.done === true ? undefined : next.value; | |
| while (pendingBatch !== undefined) { | |
| while (next.done !== true) { |
I think you can simplify the implementation and drop pendingBatch if you directly use next.done and next.value instead
Comment on lines
+306
to
+321
| const [gen, result] = async_iterator.split( | ||
| model.trainNextBatches( | ||
| datasetIterator as AsyncIterator< | ||
| Batched<DataFormat.ModelEncoded["text"]> | ||
| >, | ||
| maxBatchCount, | ||
| validationDataset as | ||
| | Dataset<Batched<DataFormat.ModelEncoded["text"]>> | ||
| | undefined, | ||
| setDone, | ||
| ), | ||
| ); | ||
|
|
||
| yield gen; | ||
| const epochLogs = await result; | ||
| epochsLogs = epochsLogs.push(epochLogs); |
Collaborator
There was a problem hiding this comment.
Suggested change
| const [gen, result] = async_iterator.split( | |
| model.trainNextBatches( | |
| datasetIterator as AsyncIterator< | |
| Batched<DataFormat.ModelEncoded["text"]> | |
| >, | |
| maxBatchCount, | |
| validationDataset as | |
| | Dataset<Batched<DataFormat.ModelEncoded["text"]>> | |
| | undefined, | |
| setDone, | |
| ), | |
| ); | |
| yield gen; | |
| const epochLogs = await result; | |
| epochsLogs = epochsLogs.push(epochLogs); | |
| const [gen, epochLogs] = async_iterator.split( | |
| model.trainNextBatches( | |
| datasetIterator as AsyncIterator< | |
| Batched<DataFormat.ModelEncoded["text"]> | |
| >, | |
| maxBatchCount, | |
| validationDataset as | |
| | Dataset<Batched<DataFormat.ModelEncoded["text"]>> | |
| | undefined, | |
| setDone, | |
| ), | |
| ); | |
| yield gen; | |
| epochsLogs = epochsLogs.push(await epochLogs); |
Comment on lines
+289
to
307
| #hashTokenContext(tokens: number[]): number { | ||
| let hash = 0x811c9dc5; | ||
|
|
||
| for (const token of tokens) { | ||
| hash ^= token & 0xff; | ||
| hash = Math.imul(hash, 0x01000193); | ||
| hash ^= (token >>> 8) & 0xff; | ||
| hash = Math.imul(hash, 0x01000193); | ||
| hash ^= (token >>> 16) & 0xff; | ||
| hash = Math.imul(hash, 0x01000193); | ||
| hash ^= (token >>> 24) & 0xff; | ||
| hash = Math.imul(hash, 0x01000193); | ||
| hash ^= 0xff; | ||
| hash = Math.imul(hash, 0x01000193); | ||
| } | ||
|
|
||
| return hash >>> 0; | ||
| } | ||
| } |
Collaborator
There was a problem hiding this comment.
Can you add documentation to this method?
Comment on lines
+302
to
+305
| const model = this.model as unknown as IterationTrainableTextModel; | ||
| if (typeof model.trainNextBatches !== "function") | ||
| throw new Error("model does not support iteration-based training"); | ||
|
|
Collaborator
There was a problem hiding this comment.
I find this part really inconvenient because it's not generic, it only addresses on particular case. Here's an idea to work around this, let me know what you think
- Implement a new dataset method that splits the dataset after a specified number of steps and returns 1) one dataset with only the next specified number of batches and 2) the remainder of the dataset after these batches. e.g., const [nextBatches, datasetRemainder] = dataset.subset(maxBatchCount)
- From runIterationRounds, I think this allows you to do reuse the existing #runRound, e.g. this.#runRound(nextBatches, roundValidationDataset)
- With this method I don't think you need to deal with all the iterators
Let me know what you think and whether this is feasible
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.