Neural networks are all the rage right now with increasing numbers of hackers, students, researchers, as well as businesses getting involved. The last resurgence was in the 80s as well as 90s, when there was little or no world wide web as well as few neural network tools. The present resurgence started around 2006. From a hacker’s perspective, what tools as well as other resources were offered back then, what’s offered now, and what should we expect for the future? For myself, a GPU on the Raspberry Pi would be nice.
The 80s as well as 90s
Neural network 80s/90s books as well as mags
For the young’uns reading this who wonder how us old geezers managed to do anything before the world wide Web, hardcopy magazines played a huge part in making us conscious of new things. as well as so it was scientific American magazine’s September 1992 special issue on Mind as well as brain that introduced me to neural networks, both the biological as well as artificial kinds.
Back then you had the option of writing your own neural networks from scratch or ordering source code from someone else, which you’d receive on a floppy diskette in the mail. I even ordered a floppy from The Amateur scientist column of that scientific American issue. You might likewise buy a neural network library that would do all the low-level, complex math for you. There was likewise a free simulator called Xerion from the university of Toronto.
Keeping an eye on the bookstore science sections did turn up the occasional book on the subject. The traditional was the two-volume Explorations in Parallel Distributed Processing, by Rumelhart, McClelland et al. A preferred of mine was Neural Computation as well as Self-Organizing Maps: An Introduction, useful if you were interested in neural networks controlling a robot arm.
There were likewise short courses as well as conferences you might attend. The seminar I attended in 1994 was a free two-day one put on by Geoffrey Hinton, then of the university of Toronto, both then as well as now a leader in the field. the best reputed annual seminar at the time was the Neural information Processing System conference, still going strong today.
And lastly, I recall combing the libraries for published papers. My stack of seminar papers as well as program handouts, photocopied articles, as well as handwritten notes from that period is around 3″ thick.
Then things went relatively quiet. While neural networks had discovered use in a few applications, they hadn’t lived up to their hype as well as from the perspective of the world, outside of a limited research community, they ceased to matter. things remained quiet as gradual improvements were made, along with a few breakthroughs, as well as then lastly around 2006 they exploded on the world again.
The present Arrives
We’re focusing on tools here however briefly, those breakthroughs were mainly:
new techniques for training networks that go more than three or four layers deep, now called deep neural networks
the use of GPUs (Graphics Processing Units) to speed up training
the availability of training data containing large numbers of samples
Neural Network Frameworks
There are now numerous neural network libraries, usually called frameworks, offered for download for free with various licenses, many of them open source frameworks. most of the more popular ones enable you to run your neural networks on GPUs, as well as are flexible sufficient to support most types of networks.
Here are most of the more popular ones. They all have GPU support except for FNN.
TensorFlow
Languages: Python, C++ is in the works
TensorFlow is Google’s latest neural network framework. It’s designed for distributing networks across multiple machines as well as GPUs. It can be considered a low-level one, offering great flexibility however likewise a larger learning curve than high-level ones like Keras as well as TFLearn, both talked about below. However, they are working on producing a version of Keras integrated in TensorFlow.
We’ve seen this one in a hack on Hackaday already in this hammer as well as beer bottle recognizing robot as well as even have an introduction to using TensorFlow.
Theano
Languages: Python
This is an open source library for doing effective numerical computations involving multi-dimensional arrays. It’s from the university of Montreal, and runs on Windows, Linux as well as OS-X. Theano has been around for a long time, 0.1 having been released in 2009.
קפה
Languages: Command line, Python, as well as MATLAB
Caffe is developed by Berkeley AI research as well as neighborhood contributors. Models can be defined in a plain text data as well as then processed using a command line tool. There are likewise Python as well as MATLAB interfaces. For example, you can define your model in a plain text file, give details on how to train it in a second plain text data called a solver, as well as then pass these to the caffe command line tool which will then train a neural network. You can then load this trained net using a Python program as well as use it to do something, image classification for example.
CNTK
Languages: Python, C++, C#
This is the Microsoft Cognitive Toolkit (CNTK) as well as runs on Windows and Linux. They’re currently working on a version to be used with Keras.
Keras
Languages: Python
Written in Python, Keras uses either TensorFlow or Theano underneath, making it easier to use those frameworks. There are likewise plans to support CNTK as well. work is underway to integrate Keras into TensorFlow resulting in a separate TensorFlow-only version of Keras.
TF Learn
Languages: Python
Like Keras, this is a high-level library built on top of TensorFlow.
FANN
Languages: supports over 15 languages, no GPU support
This is a high-level open source library written in C. It’s limited to fully connected as well as sparsely connected neural networks. However, it’s been popular over the years, as well as has even been included in Linux distributions. It’s recently shown up here on Hackaday in a robot that learned to walk using reinforcement learning, a machine learning technique that often makes use of neural networks.
לפיד
Languages: Lua
Open source library written in C. Interestingly, they say on the front page of their website that Torch is embeddable, with ports to iOS, Andoid as well as FPGA backends.
PyTorch
Languages: Python
PyTorch is relatively new, their website says it’s in early-release beta, however there seems to be a lot interest in it. It runs on Linux and OS-X as well as uses Torch underneath.
There are no question others that I’ve missed. If you have a particular preferred that’s not here then please let us know in the comments.
Which one should you use? Unless the programming language or OS is an issue then one more factor to keep in mind is your skill level. If you’re uncomfortable with math or don’t want to dig deeply into the neural network’s nuances then chose a high-level one. in that case, stay away from TensorFlow, where you have to learn more about the API than Kera, TFLearn or the other high-level ones. frameworks that highlight their math functionality usually require you to do more work to produce the network. one more factor is whether or not you’ll be doing fundamental research. A high-level framework may not enable you to gain access to the innards enough to start making crazy networks, perhaps with connections spanning multiple layers or within layers, as well as with data flowing in all directions.
שירותיים אינטרנטיים
Are you you’re looking to add something a neural network would offer to your hack however don’t want to take the time to learn the intricacies of neural networks? For that there are services offered by connecting your hack to the internet.
We’ve seen countless examples making use of Amazon’s Alexa for voice recognition. Google likewise has its Cloud machine learning services which includes vision as well as speech. Its vision service have shown up here using Raspberry Pi’s for candy sorting as well as reading human emotions. The Wekinator is aimed at artists as well as musicians that we’ve seen used to train a neural network to respond to various gestures for turning things on an off around the house, as well as for making a virtual world’s tiniest violin. Not to be left out, Microsoft likewise has its Cognitive services APIs, including: vision, speech, language as well as others.
GPUs as well as TPUs
Iterating through a neural network
Training a neural network requires iterating through the neural network, forward as well as then backward, each time improving the network’s accuracy. Up to a point, the more iterations you can do, the much better the final accuracy will be when you stop. The number of iterations might be in the hundreds or even thousands. With 1980s as well as 1990s computers, achieving enough iterations might take an unacceptable amount of time. According to the article, Deep learning in Neural Networks: An Overview, in 2004 an increase of 20 times the speed was achieved with a GPU for a fully connected neural network. In 2006 a 4 times increase was achieved for a convolutional neural network. By 2010, increases were as much as 50 times faster when comparing training on a CPU versus a GPU. As a result, accuracies were much higher.
Nvidia Titan Xp graphics card. image Credit: Nvidia
How do GPUs help? A huge part of training a neural network involves doing matrix multiplication, something which is done much faster on a GPU than on a CPU. Nvidia, a leader in making graphics cards and GPUs, created an API called CUDA which is used by neural network software to make use of the GPU. We point this out since you’ll see the term CUDA a lot. With the spread of deep learning, Nvidia has added more APIs, including CuDNN (CUDA for Deep Neural Networks), a library of finely tuned neural network primitives, as well as one more term you’ll see.
Nviדיון גם יש מחשב יחיד משלה, Jetson TX2, שנועד להיות המוח עבור מכוניות נהיגה עצמית, מזל”ט selfie-snapping, כמו גם על כך. עם זאת, כמו שלנו [benchoff בריאן] הצביע, נקודת המחיר הוא קצת גבוה עבור האקר טיפוסי.
גוגל יש גם עובד על האצת החומרה שלו בסוג יחידת עיבוד טנזור (TPU). ייתכן שתבחין בדמיון לשם המסגרת של Google לעיל, Tensorflow. Tensorflow עושה שימוש כבד של tensors (לחשוב על יחיד, כמו גם מערכים רב מימדי בתוכנה). על פי הנייר של גוגל על TPU זה מיועד לשלב ההסקה של רשתות עצביות. מסקנה מתייחסת לא להכשרת רשתות עצביות אלא להשתמש ברשת העצבית לאחר שהוכשרה. לא ראינו את זה נעשה שימוש בכל סוג של מסגרות עדיין, אבל זה משהו שכדאי לזכור.
באמצעות חומרה של אנשים אחרים
האם יש לך רשת עצבית כי ייקח הרבה זמן להתאמן אבל אין לך GPU נתמך, או לא רוצה לקשור את המשאבים שלך? במצב זה יש חומרה אתה יכול להשתמש במכונות אחרות נגיש דרך האינטרנט. אחד כזה הוא floydhub אשר, עבור אדם, עלויות רק פרוטה לשעה ללא תשלום חודשי. עוד אחד הוא אמזון EC2.
מערכי נתונים
אימון רשת עצבית עם נתונים שכותרתו
אמרנו כי אחד הפריצות דרך ברשתות עצביות היה זמינות של אימון נתונים המכילים מספר גדול של דגימות, בעשרות אלפים. אימון רשת עצבית באמצעות אלגוריתם אימונים בפיקוח כרוך נותן את הנתונים לרשת בתשומותיו, כמו כן, אומר את זה מה שהפלט הצפוי צריך להיות. במצב זה הנתונים צריכים להיות מתויגים. אם אתה נותן תמונה של סוס לתשומות של הרשת, כמו גם את התפוקות שלה אומר שזה נראה כמו ברדלס, אז זה צריך לדעת כי השגיאה היא גדולה כמו גם הדרכה יותר. התפוקה הצפויה נקראת תווית, כמו גם הנתונים הוא “נתונים שכותרתו”.
נתונים רבים כאלה מוצעים באינטרנט למטרות אימון. Mnist הוא אחד כזה עבור זיהוי אופי בכתב יד. ImageNet ו Cifar הם שני נתונים שונים של תמונות שכותרתו. הרבה יותר מופיעים בדף זה ויקיפדיה. רבים מהמסגרות המפורטות לעיל יש הדרכות הכוללות את מערכי הנתונים הדרושים.
זה לא אומר שאתה בהחלט צריך מערך נתונים גדול כדי לקבל דיוק מכובד. רובוט ההליכה שהזכרנו בעבר כי השתמשו במסגרת FNN, השתמשו במיקומי המנוע סרוו כתנו לנתוני ההדרכה שלו.
משאבים אחרים
שלא כמו בשנות ה -80, כמו גם 90, בעוד אתה עדיין יכול לקנות ספרים hardcopy על רשתות עצביות, ישנם רבים באינטרנט. שני ספרים מקוונים נהניתי הם למידה עמוקה על ידי העיתונות MIT, כמו גם רשתות עצביות ולמידה עמוקה. המסגרות המפורטות לעיל יש כל הדרכות כדי לעזור להתחיל. ואז יש אינספור אתרים אחרים כמו גם וידאו YouTube על כל סוג של נושא שאתה מחפש. אני מגלה וידאו YouTube של הרצאות מוקלטות כמו גם שיחות סמינר שימושי מאוד.
העתיד
פטל פי 7 עם GPU
ללא ספק העתיד יראה עוד מסגרות לבוא יחד.
ראינו מזמן צ ‘יפס עצביים מיוחדים כמו גם לוחות בשוק אבל אף אחד מעולם לא גילתה שוק ענק, אפילו בחזרה בשנות ה -90. עם זאת, אלה אינם מתוכננים במיוחד עבור שירות הצמיחה האמיתית, תוכנת הרשת העצבית שכולם עובדים עליהם. GPUs לשרת את השוק. כמו רשתות עצביות עם מיליוני חיבורים עבור תמונה, כמו גם עיבוד קול, שפה, כמו גם אז על לעשות את דרכם לתוך קטן יותר, כמו גם התקני צרכנים קטנים יותר הצורך יותר GPUs או מעבדים המותאמים לתוכנה זו בתקווה לגרום למשהו יכול בסופו של דבר להיות אלמנט חדש על פטל PI או לוח Arduino. למרות שיש את האפשרות כי עיבוד יישאר שירות מקוון במקום. עריכה: מתברר כי יש GPU על פטל PI – ראה את ההערות להלן. זה לא אומר כל המסגרות לעיל יעשה שימוש בזה אף. לדוגמה, Tensorflow תומך כרטיסי CUDA NVIDIA בלבד. עם זאת, אתה עדיין יכול להשתמש ב- GPU עבור קוד רשת עצבית אישית משלך. קישורים שונים הם ההערות עבור זה גם.
יש כבר תחרות עבור GPU מ Asics כמו TPU וזה אפשרי שנראה יותר של אלה, אולי דחתות GPU מ רשתות עצביות לגמרי.
באשר למחשב החדש שלנו