1)how to create my domain
Domains are organized in hierarchic structure and separate branches (.org, .org, .net and others) are managed by different organizations. You can't just create usable working internet domain on your own.
1) ... and create users with email accounts with email@example.com
from my own GUI
4)how do i build up an inbox for my users
That's the same question asked twice. This forum has scripts for account creation in script section. If you want to use your own GUI for account creation and management, you will have to create that GUI yourself by using hMailServer COM API in your language of choice. See COM API section in hMailServer documentation and learn how to use Windows COM API in your programming language.
2)what will be the smtp server name to send emails
It will be IP address or dns name of your email server. Or any other email server on your local network, which will accept your emails for relay. See http://www.faqs.org/rfcs/rfc821.html
3)can i retrieve mails coming to my users' email addresses using pop3
No. You don't know their passwords. If you do or you use other tools to read users emails without their consent, you are violating their privacy.