Spurgeon wrote this and when I read it, it rings true for what I believe about the observation and celebration of the incarnation at this time of the year:

WE HAVE NO superstitious regard for times and seasons. Certainly we do not believe in the present ecclesiastical arrangement called Christmas: first, because we do not believe in the mass at all, but abhor it, whether it be said or sung in Latin or in English; and, secondly, because we find no Scriptural warrant whatever for observing any day as the birthday of the Saviour; and, consequently, its observance is a superstition, because not of divine authority. Superstition has fixed most positively the day of our Saviour’s birth, although there is no possibility of discovering when it occurred. Fabricius gives a catalogue of 136 different learned opinions upon the matter; and various divines invent weighty arguments for advocating a date in every month in the year. It was not till the middle of the third century that any part of the church celebrated the nativity of our Lord; and it was not till very long after the Western church had set the example, that the Eastern adopted it. Because the day is not known, therefore superstition has fixed it; while, since the day of the death of our Saviour might be determined with much certainty, therefore superstition shifts the date of its observance every year. Where is the method in the madness of the superstitious? Probably the fact is that the holy days were arranged to fit in with heathen festivals. We venture to assert, that if there be any day in the year, of which we may be pretty sure that it was not the day on which the Saviour was born, it is the twenty-fifth of December. Nevertheless since, the current of men’s thoughts is led this way just now, and I see no evil in the current itself, I shall launch the bark of our discourse upon that stream, and make use of the fact, which I shall neither justify nor condemn, by endeavoring to lead your thoughts in the same direction. Since it is lawful, and even laudable, to meditate upon the incarnation of the Lord upon any day in the year, it cannot be in the power of other men’s superstitions to render such a meditation improper for to-day. Regarding not the day, let us, nevertheless, give God thanks for the gift of his dear son.

Some differ that it was a pagan festival to begin.  Maybe they’re right.  I don’t know, so I revert back to the position that Spurgeon took.