Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: refactor for Java bindings #40

Open
wants to merge 27 commits into
base: main
Choose a base branch
from
Open

WIP: refactor for Java bindings #40

wants to merge 27 commits into from

Conversation

eisber
Copy link

@eisber eisber commented Feb 28, 2023

No description provided.

eisber and others added 26 commits February 8, 2023 11:32
@eisber
refactor
6c49a41
@eisber
ported loading to Rust
09dada4
@eisber
fixed?
4c6afcb
@jackgerrits
Propagate errors from split whitespace in bpe_merges
502a6ec
@eisber
moving registry into rust
c685d99
@jackgerrits
return ref
6dcfa91
@eisber
split into multiple libs
25306a5
@eisber
fixed core
ba69bd9
@jackgerrits
Fix python setup.py for package in workspace
c98f824
@eisber
java bindings...
f248694
@eisber
more TODO
2f371f6
@jackgerrits
Initial VERY rough impl of JNI layer
1d3f707
@eisber
improve error handling
97d8dea
@jackgerrits
Improve error handling in JNI functions
53628a4
@eisber
Merge pull request #1 from eisber/error_handling
42548d8 
Improve error handling in JNI functions
@eisber
moved config into json
e3ab3f6
@eisber
add github action
22584d4
@eisber
add jar build
42efde4
@eisber
fix rust build
484f15a
@eisber
build jar
6c45bf6
@eisber
fix path
b883d0c 
add matrix java test
@eisber
fix java
98070f1
@eisber
Merge remote-tracking branch 'upstream/main' into main
3b39f6b
@eisber
cleanup
4e70da0
@eisber
update groupid
381d32b
@eisber
remove comments
f1560bd
@dqbd dqbd mentioned this pull request Mar 6, 2023
Open
@zurawiki
Copy link

These changes looks really great and useful for making tiktoken available to many different languages and runtimes.

@eisber This PR is currently titled as WIP. What about this PR is incomplete? Can we break these changes down into more digestible PRs?

@hauntsaninja (original author). Do you have any feedback here on whether these changes can be merged in / what feedback you have?

This was referenced Mar 15, 2023
Merged
Open
@jiangying000
Copy link

These changes looks really great and useful for making tiktoken available to many different languages and runtimes.

@eisber This PR is currently titled as WIP. What about this PR is incomplete? Can we break these changes down into more digestible PRs?

@hauntsaninja (original author). Do you have any feedback here on whether these changes can be merged in / what feedback you have?

Agree, Can we push this forward a little bit more? Tiktoken for java would be useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zafercavdar zafercavdar zafercavdar left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@eisber @zurawiki @jiangying000 @zafercavdar @jackgerrits