The films begins with Alfie Elkins (Caine) ending a relationship with a married woman and getting another of his affairs pregnant. The film then follows his life for a few years, documenting events that lead to the character's emotional growth starting with the birth of his child.
Watching his child grow brings out a more caring side of Alfie, including his having a health check in case of hereditary diseases, but his inability to commit to the child's mother leads to her marrying a bus conductor. The health check reveals Alfie has shadows on his lungs. This combined with being banned from seeing his son leads him to have a small breakdown.
Alfie then spends some time in a convalescence home. Here he meets Harry, who confronts him about his delusion that he is doing no harm, and Harry's wife, whom he gets pregnant in a one-night stand. The subsequent abortion is a turning point for the character and the only time other than his passing out/breakdown where he exhibits real emotion - breaking down in tears at the sight of the aborted fetus.
At the film's climax, Alfie decides to change his non-committal ways and settle down. He chooses Ruby (Shelley Winters), who is an older voluptuous and promiscuous American he met while working in taking holiday photos. Unfortunately the day he chooses to suggest this to Ruby, he finds a younger man in her bed, leaving him disheartened at the film's end and wondering “What's it all about? You know what I mean”.