<p>Hobson and Greg are working with volunteers to develop an open source
AI that we call Qary (QA for question answering). We're adding plugins
to support open source large language models (LLMs) like GPT-2 and
Llama2. Here's how you can use LLMs in your own Python Programs.</p>
<ol type="1">
<li>Create a Hugging Face account:</li>
</ol>
<ul>
<li><a href="https://huggingface.co/join">huggingface.co/join</a></li>
</ul>
<ol start="2" type="1">
<li>Create and copy your access token:</li>
</ol>
<ul>
<li><a href="https://huggingface.co/settings/tokens">Your user
profile</a></li>
</ul>
<ol start="3" type="1">
<li>Create a .env file with your access token string:</li>
</ol>
<pre class="bash"><code>echo &quot;HUGGINGFACE_ACCESS_TOKEN=hf_...&quot; &gt;&gt; .env</code></pre>
<ol start="4" type="1">
<li>Load the <code>.env</code> variables in your python script using
<code>dotenv</code> package and os.environ:</li>
</ol>
<ul>
<li>TIP: Use <code>os.environ</code> to retrieve the dict of variable
values rather than <code>dotenv.load_values</code>- Otherwise other
environment variables that have been set by other shell scripts such as
<code>.bashrc</code> will be ignored.</li>
<li>This confused us when we were getting our GitLab CI-CD pipeline
working and deploying to Render.com.</li>
<li>Each of your cloud services will have different approaches to
setting environment variables.</li>
<li>This token string can be passed as a keyword argument to most of the
pipeline and model classes.</li>
</ul>
<pre class="python"><code>import dotenv
dotenv.load_dotenv()
import os
env = dict(os.environ)
token = env[&#39;HUGGINGFACE_ACCESS_TOKEN&#39;]</code></pre>
<ol start="5" type="1">
<li>Find the path and name for the model on Hugging Face hub you want to
use:</li>
</ol>
<ul>
<li>search for "llama2" in the top search bar on <a
href="https://huggingface.co/">huggingface.co/</a></li>
<li>TIP: don't hit enter at the end of your search, instead click on
"See 3958 model results for llama2"</li>
<li>I clicked on <a
href="https://huggingface.co/meta-llama/Llama-2-7b-chat-hf">meta-llama/Llama-2-7b-chat-hf</a>
to see the documentation</li>
</ul>
<ol start="6" type="1">
<li>On the documentation page for your model you may have to apply for a
license if it's not really open source but business source like Meta
does with its AI so you can't use their models to compete with them</li>
</ol>
<ul>
<li>Apply for a license to use Llama2 on <a
href="https://ai.meta.com/resources/models-and-libraries/llama-downloads/">ai.meta.com</a>
using the same e-mail you used for your Hugging Face account.</li>
</ul>
<ol start="7" type="1">
<li>Follow the <a
href="https://huggingface.co/docs/hub/security-tokens">instructions on
huggingface.co</a> to authenticate your python session</li>
</ol>
<ul>
<li>TIP: You'll need to use the kwarg <code>use_auth_token</code> in the
<code>AutoModel.from_pretrained</code> or <code>pipeline</code>
functions.</li>
<li>And it should be set to the token from your Hugging Face profile
page. The hugging face documentation says to use the <code>token</code>
kwarg, but that never worked for me.</li>
</ul>
<pre class="python"><code>from transformers import pipeline, set_seed
generator = pipeline(&#39;text-generation&#39;, model=&#39;openai-gpt&#39;)
q = &quot;2+2=&quot;
responses = generator(
    q,
    max_length=10,
    num_return_sequences=10
    )
responses</code></pre>
<pre class="text"><code>[{&#39;generated_text&#39;: &#39;2+2= 2.2, 1.1 and&#39;},
 {&#39;generated_text&#39;: &#39;2+2= 3336 miles. they&#39;},
 {&#39;generated_text&#39;: &#39;2+2= 2, = 2 = 2&#39;},
 {&#39;generated_text&#39;: &#39;2+2= 4 = 2 = 5 \n&#39;},
 {&#39;generated_text&#39;: &#39;2+2= 0 ( 1 ) = =&#39;},
 {&#39;generated_text&#39;: &#39;2+2= 6 times the speed of sound&#39;},
 {&#39;generated_text&#39;: &#39;2+2= 2 times 5, 865&#39;},
 {&#39;generated_text&#39;: &#39;2+2= 3 / 7 / 11 =&#39;},
 {&#39;generated_text&#39;: &#39;2+2= 2 2 \n 2 of 2&#39;},
 {&#39;generated_text&#39;: &#39;2+2= 1, 9 = 1,&#39;}]</code></pre>
<p>Here's the cow leg counting question:</p>
<pre class="python"><code>    q = &quot;There are 2 cows and 2 bulls, how many legs are there?&quot;
    responses = generator(
        f&quot;Question: {q}\nAnswer: &quot;,
        max_length=30,
        num_return_sequences=10)
    answers = []
    for resp in responses:
        text = resp[&#39;generated_text&#39;]
        answers.append(text[text.find(&#39;Answer: &#39;)+9:])
    answers</code></pre>
<pre class="text"><code>     &#39;four. \n &quot; let me see if i have this straight&#39;,
     &#39;only 3. and three cows and 2 bulls are bigger than&#39;,
     &#39;2, 2, 1, 2. \n &quot; not yet&#39;,
     &quot;one per cow, that&#39;s all there is. in fact&quot;,
     &#39;30. and what am i? oh, yes, about&#39;,
     &#39;one. the big, white bull that is bigger than 1&#39;,
     &#39;three. they need to be introduced to the cow population before&#39;,
     &quot;1. i don&#39;t know how many and where exactly ;&quot;,
     &#39;no 2. 2. two bulls for 1 bull and 2&#39;,
     &#39;1, there are 1.2, and 2, there are&#39;]</code></pre>
