In recent years, energy consumption in buildings has been rising and is currently representing a significant percentage of the whole energy consumption on earth. The EU has responded this trend by requiring zero energy consumption by 2020 and by supporting innovative research approaches for improving energy efficiency in buildings without decreasing inhabitants comfort. This paper describes the approach to develop an intelligent system for building specific energy management that allows occupants and facility managers to monitor and control the energy consumption and also detects their wasting points. An ontology based information model for building energy management offering expressive representation and reasoning capability is also introduced in this paper. We highlight an approach to develop the ontology as the knowledge base providing the intelligence of the system. Furthermore we demonstrate how the energy performance analysis is improved using the ontology based approach.