-
@ someone
2024-03-19 06:28:25I realized
langchainis an overkill. Went with the ollama-python library. It connects to Ollama server and sends the query.I tried different prompt templates and it looks like llama2 is becoming a default in open source models. (The
[INST] ... [/INST]format).I played with different models, * llama2 - The OG model by Meta. Don't expect much because people iterated a lot on it and produced better ones. * llama2:70b-chat-q4_K_M - This is the 4-bit quantized version of 70 billion parameter version of the above. I.e. smarter but slower * miqu2iq - 2 bit quantization of Miqu, a leak from mistral. Thought to be Mistral Medium. * miqu4 - 4 bit quantization of the above * mixtral:8x7b-instruct-v0.1-q3_K_S - Back in the days when mistral was open sourcing, this was their last publicly open sourced model. Then they leaked Miqu. * qwen:72b-chat-v1.5-q3_K_S - another model that does well on LMSys Leaderboard. * qwen:14b - faster and stupider version of the above
I use the bold one above mostly. It has less censoring and can adapt to your directive in the system prompt.
Here is a sample run. The orange is sent by a user on Nostr. The purple text is the bot's reply.
This quick and dirty script below connects to a few relays and looks for recent notes with #AskNostr in them and produces sample replies. It does not send the replies to the network yet. It then waits for new notes that could have #AskNostr in them.
``` import queue import threading from nostr_sdk import Client, NostrSigner, Keys, Event, Filter, \ HandleNotification, Timestamp, nip04_decrypt, init_logger, LogLevel, Tag import time import ollama from colorist import rgb
def handle_kind1(q): msgs = [ {"role": "system", "content": 'You are a helper bot who is chatting with people on Nostr network.' ' People ask you questions using the #AskNostr tag.' ' You are seeing that tag and responding.' ' Your name is MIQU2.' ' Nostr is a decentralized censorship free freedom minded social media and other stuff protocol.'}, {"role": "user", "content": "Hi MIQU2, nice to meet you!"}, {"role": "assistant", "content": "Hello user, what's up!"}, {"role": "user", "content": "Hi MIQU2. What a beautiful network this is!"}, {"role": "assistant", "content": "Indeed, lots of freedom minded people!"}, {"role": "user", "content": "I am glad I joined Nostr. Plebs here are really friendly."}, {"role": "assistant", "content": "There are also friendly bots like me :)"} ]
model = 'mixtral:8x7b-instruct-v0.1-q3_K_S' # format llama2. opt = ollama.Options() opt['num_ctx'] = 8192 # load time. 8192 holds about 13 q/a's. opt['num_predict'] = 512 opt['temperature'] = 0.2 opt['repeat_penalty'] = 1.0 prompt = f"<s>[INST] <<SYS>>\n{msgs[0]['content']}\n<</SYS>>\n\n{msgs[1]['content']} [/INST] {msgs[2]['content']}</s>" i = 3 while i < len(msgs): prompt += f"<s>[INST] {msgs[i]['content']} [/INST] {msgs[i + 1]['content']}</s>" i += 2 msgs += [{"role": "user", "content": q}] prompt += f"<s>[INST] {msgs[i]['content']} [/INST] " # print(prompt) response = ollama.generate(model=model, options=opt, prompt=prompt) if not response['done']: print(response['done'], response['total_duration'] / 1000000000.0) return response['response']init_logger(LogLevel.WARN)
keys = Keys.parse("nsec1ufnus6pju578ste3v90xd5m2decpuzpql2295m3sknqcjzyys9ls0qlc85")
sk = keys.secret_key() pk = keys.public_key()
print(f"Bot public key: {pk.to_bech32()}")
signer = NostrSigner.keys(keys) client = Client(signer)
client.add_relay("wss://relay.damus.io") client.add_relay("wss://nostr.mom") client.add_relay("wss://e.nos.lol") client.add_relay("wss://nos.lol") client.connect()
kind4_filter = Filter().pubkey(pk).kind(4).since(Timestamp.from_secs(int(time.time() - 3600))) kind1_filter = Filter().kind(1).since(Timestamp.from_secs(int(time.time() - 86400))) client.subscribe([kind4_filter, kind1_filter])
evs = queue.Queue() dms = queue.Queue()
def proc_evs(): while True: ev = evs.get() event_tag_seen = False for tag in ev.tags(): a = tag.as_vec() if a[0] == "e": event_tag_seen = True # print(a) if event_tag_seen: # possibly a reply to another note return rgb(ev.content(), 247, 147, 26) response = handle_kind1(ev.content()) rgb(response, 200, 30, 255) time.sleep(30)
def proc_dms(): while True: dm = dms.get() msg = dm['msg'] author = dm['author'] rgb(msg, 247, 147, 26) response = handle_kind1(msg) rgb(response, 200, 30, 255) time.sleep(30)
class NotificationHandler(HandleNotification): def handle(self, relay_url, event: Event): if event.kind() == 4: try: msg = nip04_decrypt(sk, event.author(), event.content()) print(f"Received new DM: {msg} from {event.author()}") dms.put({'author': event.author(), 'msg': msg}) except Exception as e: print(f"Error during content NIP04 decryption: {e}") elif len(event.content()) > 20 and event.kind() == 1: # print('got new event kind 1 len(tags)=', len(event.tags())) # check if first message in thread if '#asknostr' in event.content().lower(): # ban injection attacks: banned = ['[INST]', '
', '', ''] for b in banned: if b in event.content(): print(f"Ignoring event {event.id()} due to possible injection attack") return evs.put(event)', '[/INST]', ' def handle_msg(self, relay_url, msg): returnclient.handle_notifications(NotificationHandler()) threading.Thread(target=proc_evs, daemon=True).start()
filter1 = Filter().kind(1)
events = client.get_events_of([filter1], timedelta(seconds=10))
for event in events:
print('get_events_of()', event.as_json())
while True: time.sleep(5.0)
```
There is still not much magic here. But I think interesting part will start when we train it using the Nostr wisdom.
I will also try training it using the chats with users. I.e. if it "likes" a user, it may choose to start learning from him/her. The user could say
Learn this: Cashu is a free and open-source Chaumian ecash system built for Bitcoin. Cashu offers near-perfect privacy for users of custodial Bitcoin applications. Nobody needs to know who you are, how much funds you have, and who you transact with.Then the bot in its "spare time", while not answering other users' questions, can train on those wisdom shared by Nostriches!
This whole project is about finding the "truth" and training an AI using that truth. I think Nostr is a good place to start. This is a much needed approach because it will offer people an alternative. Instead of misaligned models that are provided by big corps plebs could do their own thing..