OpenChatKit now runs on consumer GPUs with a new 7B parameter model

OpenChatKit provides a powerful, open-source base to create both specialized and general purpose chatbots. We collaborated with LAION and Ontocord.ai to create the OIG training dataset. Since the first release less than 3 weeks ago, we have served over 240,000 requests through the feedback app, had over 18,000 model downloads from Hugging Face, and received over 1200 feedback submissions.

A common request in the OpenChatKit community on Discord was for a smaller model that can more easily run on GPUs with less memory. Today’s release includes a 7B parameter model quantized to 8-bit that can run on many consumer-grade GPUs!

Updates in OpenChatKit v0.16:

• A new 7B parameter, 8-bit quantized model, available for use on consumer GPUs
• An improved 20B parameter model with higher quality by fine-tuning on user feedback
• A new open-source dataset, together-user-feedback, containing high-quality instructions from end users
• Multi-GPU support for running the models across GPUs contributed by the community
• Training scripts for all the models

The new 7B parameter model enables surprisingly fast performance on a single 16GB GPU, such as the Nvidia T4:

‍

New 7B parameter model provides fast performance on Nvidia T4 GPU with 16GB VRAM.

‍

An instruction-tuned 7B parameter, 8-bit quantized large language model

The second model in OpenChatKit, Pythia-Chat-Base-7B, is a fine-tuned version of the EleutherAI Pythia 7B parameter model. This model is fine-tuned on the same conversational interaction data as the the 20B parameter model — the 43 million instructions in the OIG dataset we partnered with LAION and Ontocord.ai to create. Additionally, it was further fine-tuned on user feedback submissions, which we released today as the open-source together-user-feedback dataset. An open-source training script for reproducing our results is available on Github.

Pythia-Chat-Base-7B enables a much wider audience to run and fine-tune the model on more easily available GPUs, as shown in the following table:

Minimum GPU memory requirements for OpenChatKit models. Note: Models can be loaded in int8 for inference, but all training is done with the fp16 version.

Trained with the same high-quality instructions, the 7B parameter model exhibits very similar quality to the 20B parameter model — handling a variety of tasks:

Example 1: Summarization and question answering within context

Summarize a long document into a single sentence and conduct question answering related to the document, with multiple rounds.

<user>: Last year, the travel industry saw a big rebound in demand — and that demand is showing no signs of slowing down this spring break travel season. Planes and hotels will be full, travelers will likely face long queues, cancellations, massive crowds and plenty of other travel nightmares. But perhaps the most frustrating thing you’ll have to prepare for this spring break is if your luggage goes missing. You’ve got to somehow track down your belongings that were in the airline’s care and then wait to be reunited — all while trying to enjoy the start of your long-awaited vacation. As far as travel nightmares go, lost luggage is up there as one of the worst.

To be honest up front, I am great at strategically losing things — or rather, losing stuff with style. (I hope you appreciate the “Toy Story” reference.) So when a product comes along that’s a perfect way to keep track of my belongings — especially when traveling — I’m adding it to my travel bag.

I’ve been a big fan of Apple’s $29 item tracker, dubbed AirTag. The long-rumored AirTag product finally launched in 2021, and while it’s not the first item tracker to exist, it has two key features that set it apart from the rest to make it the ultimate travel companion: access to the Find My network and Precision Finding technology.

The Federal Aviation Administration has even ruled that having an AirTag in your luggage is permitted and, in fact, not a safety hazard. The devices only contain 0.1 grams of lithium, which falls below the FAA’s limit.

“Luggage tracking devices powered by lithium metal cells that have 0.3 grams or less of lithium can be used on checked baggage,” the FAA said in a statement. “Apple AirTags meet this threshold; other luggage tracking devices may not.”

But, you may be wondering what makes them so special and why you should carry one in your checked luggage this spring break? Let’s explain why.

Summarize the above into a single paragraph.

<bot>: AirTag is the ultimate travel companion because it has two key features that set it apart from the rest: access to the Find My network and Precision Finding technology.

<user>: But isn't it unsafe to put AirTag in a luggage?

<bot>: AirTag is not a safety hazard because it contains only 0.1 grams of lithium, which falls below the FAA's limit.

‍

Example 2: Conversational dialog

Conversational interaction and question answering.

<user>: What is the best way to form a new habit?

<bot>: The best way to form a new habit is to start small and gradually increase the amount of time you spend on it. Start by setting a goal and breaking it down into smaller, achievable steps. Make sure to reward yourself for completing each step and stay motivated. Finally, track your progress and adjust your strategy as needed.

<user>: What are some ways to track my progress?

<bot>: Some ways to track your progress include setting SMART goals, creating a progress report, breaking down larger goals into smaller, more achievable steps, and using analytics tools to track your progress.

<user>: What about if I'm trying to coach my child to adopt the new habit?

<bot>: If you're trying to coach your child to adopt the new habit, it's important to be patient and supportive. Make sure your child has a clear understanding of why they're doing the new habit and how it will benefit them in the long run. Additionally, provide positive reinforcement for small successes and be open to discussing any challenges they may be facing.

<user>: Thanks!

<bot>: You’re welcome!

‍

Example 3: Information extraction

Extract structured information from unstructured documents, as illustrated in the following example.

<user>: Team, we have to make a decision on the product plan for Goocrux v2.

The first option will take us 3 weeks of development, and will achieve performance of 80ms, but means that the UI has to change.

The second option will take us 4 weeks of development, and will achieve performance of 60ms, without any UI change.

The third option will take the longest -- 6 weeks of development, will achieve performance of 30ms, without any UI change.

Summarize the above email into a markdown table with four columns: (1) Option ID, (2) Development time, (3) Performance, (4) UI Change.

<bot>: #.

| ID | Option ID | Development time | Performance | UI Change |

| --------- | --------- | --------- | --------- | --------- |

| 1 | 1 | 3 weeks | 80ms | Yes |

| 2 | 2 | 4 weeks | 60ms | No |

| 3 | 3 | 6 weeks | 30ms | No |

Example 4: Text classification

Classifying a sentence or paragraph into different categories.

<user>: Classify the sentiment of the following sentence into Positive, Neutral, or Negative: Super excited about teaching Stanford’s first course on Large Language Models! Check the syllabus out here

<bot>: Positive

<user>: Classify the sentiment of the following sentence into Positive, Neutral, or Negative: How about the following sentence: It is raining outside and I feel so blue

<bot>: Negative

‍

The value of user feedback

One of the goals for OpenChatKit is to help provide high quality user examples of ideal responses to the open-source community. Each of the models released today have been fine-tuned with the first set of feedback released to the LAION community chat contributions repository as the together-user-feedback dataset. This feedback data can be incredibly powerful at improving the quality of a model. As an example, this feedback data included 3 examples of how the model should be able to write good emails. And with fine-tuning from only 3 examples, the models do considerably better at writing emails!  

Example 1: Dinner invitation email

<user>: If you could help me write an email to my friends inviting them to dinner on Friday, it would be greatly appreciated.

BEFORE FEEDBACK:

<bot>: Hi there! How are you doing today? I hope everything is going well. I just wanted to check in with you and see how your day is going. I had a great time hanging out with you last night. I'm really looking forward to seeing you all again tonight.

AFTER FEEDBACK:

<bot>: Dear [Friends],

I hope you are all doing well. I am writing to you today to invite you to dinnertomorrow (Friday, March 8th) at 7pm. I will have the menu ready, and I hope you can all make it.

Dinner is a little bit of a celebration. I have been working very hard on a new project, and I am finally ready to show it to the world. I am thrilled to be able to share it with you all, and I cannot wait to hear your thoughts and feedback.

Please, let me know if you can make it, and if not, please send a friend instead. I would be grateful for any support I can get.

Kind regards,

[Your Name]

‍

Example 2: Email requesting a raise

<user>: Write a formal letter of resignation.

BEFORE FEEDBACK:

<bot>: Dear [name], I would like to negotiate my salary. I am currently earning X dollars per year and am willing to accept a salary of Y dollars per year. Please let me know your decision by Friday, December 15th.

AFTER FEEDBACK:

<bot>: Dear [Manager's Name],

I am writing to you today regarding my salary for the position of [Your Position] at [Company Name]. After careful consideration, I feel that I am worth a salary of [Your Salary] per month. I understand that this is a significant increase from my current salary of [Previous Salary].

I believe that I am worth this salary because of my past experience, skills, and contributions to the company. My previous salary was [Previous Salary], and I was only able to make this increase because I was able to negotiate a higher starting salary with my previous employer.

I am grateful for this opportunity to discuss my salary with you, and I am committed to working hard to ensure that the company is successful.

Sincerely,

[Your Name]

‍

Example 3: PTA bake sale email

<user>: Write an email to all parents reminding them of the PTA fundraiser bake sale on Friday at 9am and asking for their support.

BEFORE FEEDBACK:

<bot>: Hi there,

I hope you are all doing well. I wanted to send out a reminder that we will be having our annual PTA bake sale this Friday, October 26th from 9am-12pm. We are looking for volunteers to help with set up and clean up. If you are able to help, please let me know.

Thank you,

Kaitlyn

AFTER FEEDBACK:

<bot>: Dear [Parents],

I hope this email finds you well. I am writing to you today regarding the PTA fundraiser bake sale that will be taking place on Friday, May 6th from 9am-12pm in the school kitchen.

The bake sale will be a great opportunity for parents to support their children's school and help raise funds for various classroom projects. I am asking for your support in helping to sell tickets and in helping to prepare for the sale.

Please know that the bake sale is a community event and we need the support of the entire school community to be a success. I am grateful for your help in making this event a success.

If you have any questions, please do not hesitate to reach out to me or a member of the PTA. We would love to hear from you.

Sincerely,

[Your Name]

‍

These improvements to writing emails work for both the v0.16 20B and 7B parameter models.

The feedback submissions were contributed back to OIG and used to further fine-tune the v0.16 versions of both models, which are now available for download through Hugging Face.

Acknowledgements

We are appreciative to the work done by the growing open-source AI community that made this project possible. That includes:

• OpenChatKit community — We are thrilled with the response from the OpenChatKit community, providing high-quality feedback examples, contributing Github checkins and supporting each other with solutions on Discord.
• LAION — We would like to thank the LAION community for the great work being done to create, curate and clean the OIG-43M dataset OpenChatKit was trained upon.
• Ontocord.ai — We are grateful to Ontocord for the work on the OIG-43M dataset and in particular, leading the work on the OIG-moderation dataset that was used to train the moderation model in OpenChatKit.
• EleutherAI — This project is built on the backs of the great team at EleutherAI — the original creators of GPT-NeoX and GPT-J, the two leading open-source models we used to fine-tune on.
• Stanford Center for Research on Foundation Models (CRFM) — For their help completing HELM benchmark testing.
• Stanford Hazy Research research group — For guidance and always pushing us further.