Neural Networks (aka Inceptionism)

[lpetrich]

The failure to create human-level AI has been a great disappointment for me.

But it looks like we are a little bit there.

Artificial intelligence has had two main approaches: top-down, by explicit specification, and bottom-up, by learning. The top-down approach seemed to be the way to go at first, but the bottom-up approach has increased in popularity over the last few decades because of ever-increasing computing power.

What Google is doing with those images is essentially bottom-up learning. Those surreal dog-face ones were likely made with a module trained on recognizing dog faces, so that module tends to see a dog face wherever it looks.

[lisarea]

I've done some of the top-down stuff, and I don't think anyone has ever seriously believed that the prescriptive approach in itself was going to result in strong AI. It was, and still is, used in narrow AIs, though.

I get the impression sometimes that a lot of people think there's going to be some big invention or discovery and all of a sudden, we'll have a single, fully functioning strong AI announced in all the headlines or something, like TA-DA singularity!, but it's probably not going to happen like that. People have been working on this for a long, long time, building on previous work and advancements. I am not all that up on how learning systems work and what progress has been made, but they are cumulative, and they do seem, at least to me, to be accelerating very quickly right now. I mean, yeah, recognizing and mimicking dog faces isn't a super-important thing or anything, but it's really mindblowing that they can do that. (Fuck Google, still. Doesn't make up for everything else, just so we're all clear on that.)

I suspect we're a lot further off and also a lot closer than most people realize. And don't discount some of the robust narrow AIs that are already out there. Everyone's distracted fretting about philosophimical questions about consciousness and stuff, but it's probably narrow AIs that are going to destroy everything you know and love.

[Ensign Steve]

FYI, I'm taking a two-day "crash course" on Tensor Flow at work tomorrow and Friday. I don't really have anything going on this weekend, so I'll probably make the singularity happen Saturday or Sunday. Or I'll get caught up on Game of Thrones. We'll see what I'm in the mood for..

[lpetrich]

lisarea, that's likely correct. But a problem with a bottom-up approach is that it can be difficult to interpret what its parameter values mean.

The history of AI is rather interesting.
History of artificial intelligence - Wikipedia
Timeline of artificial intelligence - Wikipedia
Progress in artificial intelligence - Wikipedia
Applications of artificial intelligence - Wikipedia

AI was speculated on for centuries, and I think that the culmination of such speculations was Alan Turing's classic paper Computing Machinery and Intelligence A.M. Turing

He proposed the Turing Test in it. In present-day terms, it is whether it is possible to write a chatbot that gives responses indistinguishable from a human interlocutor's responses.

Something like this:

Q: Please write me a sonnet on the subject of the Forth Bridge.
A : Count me out on this one. I never could write poetry.
Q: Add 34957 to 70764.
A: (Pause about 30 seconds and then give as answer) 105621.
Q: Do you play chess?
A: Yes.
Q: I have K at my K1, and no other pieces. You have only K at K6 and R at R1. It is your move. What do you play?
A: (After a pause of 15 seconds) R-R8 mate.

The addition result is incorrect: it's really 105721. Alan Turing was imagining simulating human thought processes, erroneous result and all.

The chess notation is old descriptive notation, something that's gone out of style. Nowadays, we use algebraic notation: I have my king at e1 and no other pieces. You have your king at e3 and your rook at h8, and also no other pieces. What move do you make? The answer: Rh1 checkmate.

Alan Turing also considered several counterarguments to the possibility of human-level AI, like a theological one about souls, "heads in the sand", Goedel's theorem, informality of behavior, etc.

-

Not long after he wrote his article, actual AI programming started. In 1951, some programmers wrote programs for early computers that could play checkers and chess.

In 1956 was a famous conference on AI at Dartmouth College in New Hampshire. The proposal for it included this assertion: "every aspect of learning or any other feature of intelligence can be so precisely described that a machine can be made to simulate it".

AI had some successes for the next 18 years, like being able to solve high-school-algebra word problems and being able to understand what goes on in "microworlds", like sets of blocks that can be moved around and stacked.

Some AI advocates were optimistic enough back then to expect human-scale AI in the next few decades.

-

But in 1974, the first "AI winter" happened. Funding agencies were reluctant to finance much work in AI, and this was because AI research was not delivering anything close to its more optimistic expectations. Part of it was that the expecters rather grossly underestimated how difficult AI would be. Part of it was the enormous amount of computing power needed for some applications, like artificial vision, and part of it was the huge quantities of data that are "common sense" for us. Both were far beyond the computers of the 1950's and 1960's, though they are not as far nowadays.

-

Then in 1980, AI funding came back, with such successes as expert systems and artificial neural networks.

But then, in 1987, another AI winter, another funding drought.

Then in 1993, AI came back again, and it has continued to the present without another AI winter.

There are at least two differences:

• AI has proven to be much more useful than previously.
• AI researchers often don't call it AI.

[lpetrich]

Originally Posted by JoeP

Now. Srs bzns: read this. The Unreasonable Effectiveness of Recurrent Neural Networks ...

Interesting.

I've done something similar by creating statistical models. Essentially Markov-chain models. Let's say that I want to generate random words that resemble the words in some corpus, like a dictionary.

A simple approach would be to make a table of the frequency of each letter, and include an end-of-word symbol with the letters. One then generates a word by choosing a letter at random according to this frequency table, and continuing until one gets an end-of-word symbol.

That tends to produce nonsense, so a better approach is to look back to the previous letter, and construct a current-letter frequency table for each previous letter found. One also needs a beginning-of-word symbol here. Then, when generating a word, one looks back to the previous letter and uses its frequency table. It generates more reasonable-looking words, because it makes vowels tend to follow consonants and vice versa, and also gets vowel and consonant clusters right.

One can extend this approach to more than one letter, though one ought to make current-letter frequency tables for each partial sequence of previous letters as well as each complete sequence. That would take care of the case of not being able to look back all the way.

Here are some results. The * is for a word that was used for setting up the lookbacks and the frequency tables.

Lookback: 0
oa
olsrfipioiomnotuegonrryeean
oo
e *
rueampruecit
eaetnymicrkoraeoeefalina
igcsaa
aesoce
e *
itoi

Lookback: 1
pa *
ontolionomminntuleriroukean
pm
c *
qulanorulbis
dagun
medrioragndelanera
heduca
betice
c *

Lookback: 2
paver *
pershomless
tensly
undal
pote *
quadiroural
unagon
meess
palteleallocyped
abucetoff

Lookback: 3
paunirreligilling
telliopterapy
pse
quadrinshakroman
mulic
intaghainamend
hebracylistely
unsol
collingly *
lusia

Lookback: 4
cacoperilability
gant *
poodlessness
gilt *
nutcraft
archite
moraceously
antproof *
overdish
disturbed *

Lookback: 5
paunch *
enneahedron *
denotoxicator
unwithdraw
delphian *
multiramolendidad
nazaretta
barnaby *
unsolvability
pretractedly


I've also created a similar Random Sentence Maker.

[lpetrich]

One of the first proposed AI applications was translating natural languages, but that turns out to be much more difficult than expected. One can come up with lots of rules, but there are usually lots of exceptions to them, so an explicit rule-based approach is very difficult.

An alternative is statistical translation, but that requires large amounts of already-translated text. Google uses text from the UN, the EU, and other such sources.

It also involves a *lot* of number crunching, a *lot* more than what was feasible in the early decades of computers.

Statistical machine translation - Wikipedia, used by Google and Bing.

Google Translate - Wikipedia -- it now supports 103 languages, with 14 in the works.

Bing - Wikipedia, the free encyclopedia -- its translator is no slouch either.

[lpetrich]

Google's image search has an interesting feature: search for similar-looking images. That must require some artificial vision, like preparing a sort of summary of an image for quick initial comparisons.

[But]

Originally Posted by But

This dog-face-eyes stuff is getting boring. It's now time to move on to pornographic images.

open_nsfw

show spoiler

[thedoc]

The real images were interesting, the altered images were fucked up, get over it.

[erimir]

I like how the concert series replaces the musicians with giant cocks. Seems appropriate.

[But]

Originally Posted by thedoc

The real images were interesting, the altered images were fucked up, get over it.

What real images?

[erimir]

So now there's a website where you can make your own images using neural networks based on Deep Dream.

One of the options is to give it a "style" image, and it will recreate a target image in the "style" (roughly speaking) of a chosen image (or some preset styles).

For example, here's the logo in three styles:

[JoeP]

Dammit there goes the rest of the day

[JoeP]

Imgur

Embedding might work:

[BrotherMan]

[lisarea]

I think it's a selfie.

[JoeP]

Originally Posted by BrotherMan

[JoeP]

Trip on LSD without the LSD! Have fun

[Dragar]

Interesting paper.

Despite the success of neural networks (NNs), there is still a concern among many over their "black box" nature. Why do they work? Here we present a simple analytic argument that NNs are in fact essentially polynomial regression models. This view will have various implications for NNs, e.g. providing an explanation for why convergence problems arise in NNs, and it gives rough guidance on avoiding overfitting. In addition, we use this phenomenon to predict and confirm a multicollinearity property of NNs not previously reported in the literature. Most importantly, given this loose correspondence, one may choose to routinely use polynomial models instead of NNs, thus avoiding some major problems of the latter, such as having to set many tuning parameters and dealing with convergence issues. We present a number of empirical results; in each case, the accuracy of the polynomial approach matches or exceeds that of NN approaches. A many-featured, open-source software package, polyreg, is available.

[JoeP]

Do we need to understand

polynomial regression models
convergence problems
overfitting
multicollinearity ()

to have a chance of understanding this 28 page paper?

[Limoncello]

[But]

Originally Posted by Dragar

Interesting paper.

Despite the success of neural networks (NNs), there is still a concern among many over their "black box" nature. Why do they work? Here we present a simple analytic argument that NNs are in fact essentially polynomial regression models.

Very, very simple indeed, and everything is "essentially" everything else. Too much woolly prose for my taste. Equations are suspiciously absent.

As a toy example, take the activation function to be a(t) = t^2. Then outputs of that first layer will be quadratic functions of u and v.

No shit.

Toy example = I can't rigorously show anything beyond that, but I have to publish this paper, so there you go.

Now let’s turn to more realistic activation functions, we note that they themselves can usually be approximated by polynomials.

Holy shit. Seriously? So in physics, everything is a harmonic oscillator because the potential can be approximated by a parabola.

For general activation functions and implementations, we can at least say that the function is at close to a polynomial, by appealing to the famous Stone-Weierstrass Theorem [Katznelson and Rudin(1961)], which states that any continuous function on a compact set can be approximated uniformly by polynomials.

The informal arguments above could be made mathematically rigorous.

In that manner, we can generate polynomials which are dense in the space of regression functions. But let’s take a closer look.

Yeah, we could make this rigorous, but instead let's take a closer look.

That's what I hate about a lot of computer science papers - a lot of prose with lots of excuses and then we run a program until we get nice numbers and dump the logs in the paper.

[Dragar]

Originally Posted by But

So in physics, everything is a harmonic oscillator because the potential can be approximated by a parabola.

Sidney Coleman once remarked that the career path of a theoretical physicist is solving the simple harmonic oscillator in ever increasing levels of abstraction.

It reminded me a lot of this much earlier paper, which you may prefer.

https://projecteuclid.org/euclid.ss/1177010638