Well then you're in good company!
The great, esteemed Apostle, the "Rock" upon which Jesus would build his church, Peter, was plagued by this.
He denied knowing Jesus during Jesus' trial and subsequently, we read in Galatians, Paul has to take him to task again for playing to the crowd (Galatians 2:11 – 14). In this story Paul clearly states that Peter was "afraid" of what a certain group of people might think of him. And Paul also clearly states that Peter was in the wrong.*
This story about Peter exemplifies the main point of Paul's letter to the Galatians:
A quick glance through the letter will reveal a recurring theme of the men versus God. In his introduction Paul makes an exclusive qualification of his Apostleship, the only time he does this in any of his letters. He says that he has NOT been appoint by men or sent by men, but by God. He is making his point right from the start. Later on he asks the rhetorical question "am I trying to impress men or God?".
Paul is making the point throughout the letter that we are not to live to impress people or for their acknowledgement.
But in making this point, Paul is not giving us license to be "individualistic". He's not saying that we can have our own little secret Christianity where no one can interfere. On the contrary, Paul says later on in the letter that we must gently correct those that we see behaving in error (6:1). You can't have your own little private communion with God irrespective of other Christians.
Paul's point is that we should never be afraid of showing the Fruits of the Spirit (5:22-25), regardless of the circumstances and regardless of what people might think of us. Peter should never have separated himself from the Gentiles for fear of the circumcision group. He should have maintained unity with his fellow Christians regardless of what the circumcision group thought of him.
The thing that transcends human systems is Jesus Christ. Paul says as much to Peter when he explains that both Jews and Gentiles are saved through faith in Jesus Christ. Our union with Him supplies us with the Spirit though whom we receive the fruits of the Spirit: love, joy, peace, patience, etc. Our union with Him also binds us together with a bond that transcends human boundaries, human systems, and human hierarchies. In Jesus we are free from humanistic restraints and attitudes.
The choice is to live with Jesus or to live for people.
You can't do both.
* Some have used this passage to show the disunity between Paul and Peter and thereby undermine Paul's credibility. But the reason Paul uses this story is to illustrate his main point. He's not trying to put down Peter in any way, or grind an axe. Later in the letter we see Peter and other esteemed apostles had extended to Paul the "right hand of fellowship", no small thing. In addition, far from undermining Paul's credibility, this story reveals the fallibility of Peter.